👌 Fix quadratic complexity in fragments_join / text_join - #389

…cters

Long runs of characters that begin an inline rule but never complete a
construct -- bare `&`, `<`, `~`, `{`, incomplete entities -- were tokenised
in O(n^2) time, so a few hundred KB of such input stalls `render()` for
many seconds. Output was always correct; the cost is pure CPU.

Two independent quadratic factors, each hit once per character in a run:

1. `state.pending` accumulation. The inline tokenizer's fallback path did
   `state.pending += state.src[state.pos]` one character at a time. Because
   `pending` is an *attribute*, `str += ch` cannot use CPython's in-place
   concatenation optimisation (the attribute holds a second reference), so
   every append copies the whole accumulated string.

2. `src[pos:]` slicing in `entity` / `html_inline`. Both matched
   `^`-anchored regexes against `state.src[pos:]`, copying the remainder of
   the source on every `&` / `<`. On Python 3.10 this carried a second
   quadratic factor inside the regex engine itself: before 3.11, `search()`
   on a `^`-anchored pattern retries at every offset.

Neither is quadratic in the JavaScript markdown-it (rope-backed string
concatenation, and sticky/lastIndex regex matching), so this is specific to
the Python port.

Fix:

- `StateInline.pending` now accumulates through a lazily-materialised list
  buffer via `append_pending()`, giving amortised O(1) appends. It is still
  exposed as a plain `str` property, so existing readers and third-party
  plugins doing `state.pending += x` keep working unchanged.
- `entity` and `html_inline` anchor with `.match(state.src, pos)` instead of
  slicing; the leading `^` is dropped since `.match` already anchors at
  `pos`. Neither pattern uses `\b` or lookbehind, so this is exactly
  equivalent. `HTML_TAG_RE` has a single consumer; the separate
  `HTML_OPEN_CLOSE_TAG_RE` is untouched.

All affected inputs now grow ~2.0x per doubling (linear) out to 640k
characters. `"&" * 400_000` drops from 6.8s to 1.3s on CPython 3.11, from
653s to 1.8s on CPython 3.10, and from 145s to 0.2s on PyPy. Runs of `[`
were also reported as superlinear, but once the `pending` quadratic is
removed they measure a flat 2.0x per doubling: that path is linear already
(bounded by the existing `skipToken` cache), just with a large constant.

Output is unchanged: a 47,936-case differential run over the repo fixtures,
the CommonMark spec, targeted constructs and fuzzed inputs across seven
presets compares rendered HTML, `renderInline`, and full token streams
byte-for-byte against the previous behaviour with zero differences. The
test suites of mdit-py-plugins, myst-parser, mdformat and rich are
unaffected.

Adds `test_long_special_char_runs_are_linear`, which asserts correct output
for the affected constructs and renders inputs that took ~16s before this
change, so a regression trips the global 10s test timeout.

Same class as the previously fixed #367 and #389. The root causes and this
fix were independently identified by Haim Dimer in #411, which this
supersedes.

Co-authored-by: Haim Dimer <haim@dimer.org>

The two regression tests added with the fix relied on the global 10s
pytest timeout to catch a return to quadratic behaviour. CI runs every
matrix job under `--cov`, whose tracer slows these pure-Python loops by
more than 4x, so the tests timed out on the fixed code.

Following the precedent of #367 and #389, the tests now pin the
behaviour deterministically rather than by wall clock:

- `test_inline_rules_do_not_slice_remaining_source` drives the inline
  parser with a `str` subclass that records every slice taken from it.
  The old `entity` / `html_inline` rules produced ~5000 slices of up to
  10000 characters on a 5000-opener run; the only slices left are the
  `text` rule's single-character chunks.
- `test_pending_appends_do_not_materialise` checks that `append_pending`
  buffers fragments without building the string until `pending` is read.
- `test_long_special_char_runs_render_correctly` keeps the output
  assertions for every affected construct at modest sizes.
- `test_rule_reading_pending_each_char_renders_correctly` interleaves
  per-character reads of `pending` with appends and checks the output.

Each of the structural tests fails on the pre-fix code and passes in well
under a second with or without coverage.

The two regression tests added with the fix relied on the global 10s
pytest timeout to catch a return to quadratic behaviour. CI runs every
matrix job under `--cov`, whose tracer slows these pure-Python loops by
more than 4x, so the tests timed out on the fixed code.

Following the precedent of #367 and #389, the tests now pin the
behaviour deterministically rather than by wall clock:

- `test_inline_rules_do_not_slice_remaining_source` drives the inline
  parser with a `str` subclass that records every slice taken from it.
  The old `entity` / `html_inline` rules produced ~5000 slices of up to
  10000 characters on a 5000-opener run; the only slices left are the
  `text` rule's single-character chunks.
- `test_pending_appends_do_not_materialise` checks that `append_pending`
  buffers fragments without building the string until `pending` is read.
- `test_long_special_char_runs_render_correctly` keeps the output
  assertions for every affected construct at modest sizes.
- `test_rule_reading_pending_each_char_renders_correctly` interleaves
  per-character reads of `pending` with appends and checks the output.

Each of the structural tests fails on the pre-fix code and passes in well
under a second with or without coverage.

## Summary

Long runs of characters that begin an inline rule but never complete a
construct (bare `&`, `<`, `~`, `{`, incomplete entities, …) were
tokenised in **O(n²)** time, so a few hundred KB of such input stalls
`render()` for many seconds. Output was always correct; the cost is pure
CPU. This is the same class as #367 and #389.

Supersedes #411. The root causes and the core fix were independently
identified by @hdimer there, and the first commit here is code-identical
to that PR (credited as co-author, thank you!). The second commit closes
a gap found while auditing it that made the fix ineffective for a common
plugin configuration, and adds a few hardening changes.

## Root causes

Two independent quadratic factors, each hit once per character in a run:

1. **`state.pending` accumulation.** The inline tokenizer's fallback
path did `state.pending += state.src[state.pos]` one character at a
time. Because `pending` is an *attribute*, `str += ch` can't use
CPython's in-place concatenation optimisation (the attribute holds a
second reference), so every append copies the whole accumulated string.
2. **`src[pos:]` slicing in `entity` / `html_inline`.** Both matched
`^`-anchored regexes against `state.src[pos:]`, copying the remainder of
the source on every `&` / `<`. On Python 3.10 this carried a *second*
quadratic factor inside the regex engine: before 3.11, `search()` on a
`^`-anchored pattern retries at every offset.

Neither is quadratic in the JavaScript markdown-it (rope-backed concat,
sticky regex matching), so this is Python-port-specific.

## Fix

**Commit 1** (as in #411):
- `StateInline.pending` accumulates through a lazily-materialised list
buffer via a new `append_pending()` method, giving amortised O(1)
appends. It's still exposed as a plain `str` property, so existing
readers and plugins doing `state.pending += x` keep working.
- `entity` and `html_inline` anchor with `.match(state.src, pos)`
instead of slicing; the leading `^` is dropped since `.match` anchors at
`pos`. Neither pattern uses `\b` or lookbehind, so this is exactly
equivalent.

**Commit 2** (new):
- The buffered getter materialised with `self._pending += joined`,
itself an attribute concat. Any rule that reads `state.pending` before
its cheap character check therefore re-introduced the quadratic on every
character. The mdit-py-plugins `attrs` rule does exactly that, and
myst-parser's `attrs_inline` extension enables it: with it, `"~a~" *
400_000` still took **34 s** (5.7× per doubling) after commit 1. The
getter now moves the string into a local and drops the instance's
reference before concatenating, so CPython resizes it in place: **3.0 s,
2.0× per doubling**, verified on CPython 3.10–3.13.
- The setter no longer depends on `__init__` having run (a subclass
assigning `pending` before `super().__init__()` raised
`AttributeError`).
- `copy.copy(state)` no longer shares the mutable buffer with the
original.

## Results

`"&" * 400_000`, before → after:

| interpreter | before | after |
|---|---|---|
| CPython 3.10 | 653 s | 1.8 s |
| CPython 3.11 | 6.8 s | 1.3 s |
| CPython 3.13 | 6.6 s | 1.1 s |
| PyPy 7.3 | 145 s | 0.2 s |

All affected inputs now grow ~2.0× per doubling out to 640k characters.
Runs of `[` were also reported as superlinear, but once the `pending`
quadratic is removed they measure a flat 2.0× per doubling: that path
was already linear (bounded by the `skipToken` cache), just with a large
constant.

**Trade-off:** the property indirection costs ~2% on ordinary Markdown
(up to ~4.7% on inline-dense files) — measured over 55 inputs × 3
presets, 8 interleaved rounds. The regex half is a pure win at every
size. Memory is unchanged on realistic input. If wanted, most of the 2%
can be recovered by having `push()`/`pushPending()` read the private
fields directly; happy to do that as a follow-up once benchmarked
properly.

## Verification

- **Output unchanged:** 47,936-case differential (repo fixtures,
CommonMark spec, targeted constructs, 6k fuzzed inputs × 7 presets)
comparing HTML, `renderInline` and full token streams byte-for-byte: 0
differences.
- **Downstream:** mdit-py-plugins (511), myst-parser (1245), mdformat
(4282), rich and mdformat-gfm test suites give identical results against
this branch and `master`; a 61-document differential through a full
plugin stack shows 0 differing renders.
- Full suite passes on CPython 3.10, 3.11, 3.12, 3.13 and PyPy. Pinned
`ruff` and strict `mypy` clean.
- New tests: `test_long_special_char_runs_are_linear`,
`test_state_inline_pending_buffer_semantics`,
`test_pending_reader_rule_stays_linear`. Each linearity test exceeds the
global 10 s timeout on the pre-fix code.

## Notes for reviewers

- `HTML_TAG_RE`, `DIGITAL_RE` and `NAMED_RE` lose their `^` anchor
(they're now applied with `.match(src, pos)`). No external consumer was
found in any scanned package or via code search, but any third-party
`.search()` on them would now be unanchored. Probably worth a changelog
line. `HTML_OPEN_CLOSE_TAG_RE` is untouched.
- mdformat-gfm ships its own replacement `text` rule that still does
`state.pending +=`; it keeps its quadratic and gains nothing from this
fix.
- **Not covered here, follow-up PR:** with `html=True` (the `commonmark`
and `gfm-like` presets), runs of `<![CDATA[`, `<!--`, `<?` and `<!a` in
inline context are still quadratic because the regex sub-patterns rescan
to end-of-input on every attempt (`<![CDATA[` × 40k takes ~245 s). This
is inherited from upstream (it's quadratic in markdown-it JS too),
pre-existing, and a different code path, so it's kept separate. A
validated fix (terminator quick-reject, zero output change) is ready.
- The changelog is left for the release, following prior PRs.

---------

Co-authored-by: Haim Dimer <haim@dimer.org>